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Abstract 



An associative memory has been discussed of neural networks consisting of spiking N{= 100) 
Hodgkin-Huxley (HH) neurons with time-delayed couplings, which memorize P patterns in 
their synaptic weights. In addition to excitatory synapses whose strengths are modified after 
the Willshaw-type learning rule with the 0/1 code for quiescent /active states, the network 
includes uniform inhibitory synapses which are introduced to reduce cross-talk noises. Our 
simulations of the HH neuron network for the noise-free state have shown to yield a fairly 
good performance with the storage capacity of ac = Pma.x/N ~ 0.4 — 2.4 for the low neuron 
activity of / ~ 0.04—0.10. This storage capacity of our temporal-code network is comparable 
to that of the rate-code model with the Willshaw-type synapses. Our HH neuron network is 
realized not to be vulnerable to the distribution of time delays in couplings. The variability 
of interspace interval (ISI) of output spike trains in the process of retrieving stored patterns 
is also discussed. 
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I. INTRODUCTION 



The Hopfield-type neural networks have been intensively investigated as a model for 
the learning and memory of brain [|I| . In this type of models the information is stored as 
a content-addressable memory in which synaptic strengths are modified after the Hebbian 
rule. Its retrieval is made when the network with the symmetric couplings works as the 
point-attractor with the fixed points. These models have been analyzed by the method 
of statistical mechanics of spin glasses 0. Such analysis provides us with most important 
information of storage capacity, role of noise and recall performance. The Hopfield-type 
models, however, are an extreme abstraction of real neural networks and they have the 
following issues from the biological viewpoint. 

(i) The dynamical variable in the Hopfield-type model is the firing rate of neurons {rate 
coding). However, the importance of a precise timing of firing {temporal coding) is currently 
realized in many experiments: sonar processing of bats f^, sound localization of owls 0, 
electrosensation in electric fish 0, visual processing of cats |^ [|^, monkeys |^ and human 

i- 

(ii) The synaptic weight Wjk for a given synapse between the pre-synaptic neuron k and 
the post-synaptic neuron j in the Hopfield-type models is usually assumed to be given by 



W,k = T.i^^f-a){d'^-b), (1) 

where P stands for the number of patterns to be stored, ^j'^^= -1/1 (or 0/1) for quies- 
cent/active states and a = b = f =< ^j^^ >, the average activity. Equation (1) shows 
that a given synapse may be positive (excitatory) or negative (inhibitory) depending on the 
covariance of the pre- and post-synaptic activities. This is not biologically realized because 
the synapse is either excitatory or inhibitory with little exception. Furthermore, Eq.(l) 
shows that the potentiation of the synaptic strength of a given synapse is possible even in 
the absence of pre- and post-synaptic activities (^j^^ = = — 1 or 0). The potentiation 
without the brain activity again contradicts the biological evidence. 

In recent years some theoretical studies have been reported of the learning and memory 
in terms of the temporal coding with the use of spiking neurons |T^- [|1^] or oscillating 
neurons |17]- |21|. It has been shown that spiking neurons such as the spike- response and 
integrate-and-fire neurons with a correlation-based Hebbian learning rule can operate very 
fast in the order of ten milliseconds |Tl| . Mueller and Herz |]15[ showed that the networks 



consisting of spiking integrate-and-fire (IF) neurons possess the storage capacity similar to 
those of rate-code neurons. Their model, however, adopts the leaning rule given by Eq.(l) for 
storing patterns in synapse weights, and then it has the second issue raised above. Based on 
spiking FitzHugh-Nagumo (FN) neurons with time-delayed couplings, Yoshioka and Shiino 
p!3| , and Kanamuru and Okabe discussed the associative memory with the use of the 



leaning rule given by Eq.(l) but with a = and b = f for the 0/1 code. Their approach 
has the second issue because their Wjk may be positive for negative depending on whether 
= 1 or for ^ = 1. 

Among spiking neuron models having been proposed so far, the Hodgkin-Huxley (HH) 
model is expected to be the most realistic in the biological sense. Since the HH model was 
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proposed in 1952 |22], its property has been intensively investigated. Its responses to applied 



dc 1 2^ and sinusoidal currents [24|, and spiking impulses have been studied. The HH 



model has been generalized with modifications in ion conductances and widely adopted for 
a study of biological systems such as hippocampus |2^ and thalamus . 



Quite recently, Lytton [|T^ has proposed the feedfoward hetero-associative memory net- 
work of spiking 40 HH neurons, by employing the notion of the Anderson-Kohonen network. 
The weight of the excitatory synapses is assumed to be given by the conventional Hebbian 
rule: 

w,,=j:^^f^it\ (2) 

with = 0/1 for quiescent /active states. In addition to the excitatory synapses, Lytton has 
included inhibitory synapses whose weights are determined by a feedfoward rule depending 
on input patterns, in order to suppress the cross-talk noises arising from the non-orthogonal 
inputs. Equation (2) shows that the synapse is strengthen only when both pre- and post- 
synapses are active, in agreement with biological experiments. It is shown, however, that the 
retrieval of the HH network is only possible in the narrow parameter ranges of conductances 
of excitatory and inhibitory synapses |]T2| . 



Since a naive transplant of the Hebbian rule given by Eq.(2) to HH neuron networks 
yields the poor performance in retrieving memorized patterns, as was shown by Lytton ||12|| , 
we adopt, in this paper, the Willshaw rule given by 

w,,= e(f: ei'^)), (3) 

where 9(x) = 1 for x > and otherwise, and = 0/1 for quiescent/active states. The 
synaptic weight given by Eq.(3) is either one or zero, in contrast with that given by Eqs. 
(1) and (2). Such a dramatic reduction is not biologically justifiable except under some 
conditions when the postsynaptic membrane of a dendritic spine is active |3^. It has been. 



however, reported that the Willshaw rule given by Eq.(3) improves the storage efficiency 



of Hopfield-type neural networks p8[- |Q. Equation (3) shows that the synaptic weight is 
potentiated only for the simultaneous firings of the pre- and post-synaptic neurons, which 
is in accord with biological results. 

Theoretical studies on memory have extensively pursued the analytical approach, requir- 
ing simplified tractable models, which do not necessarily refiect the aspect of the biological 
reality. As an alternative approach, we may take a simulation method for a better under- 
standing of functioning of real, biological systems to which an analytical method cannot 
applied. 

It is the purpose of the present paper to construct biologically plausible model for an as- 
sociative memory. We propose HH neuron networks with excitatory and inhibitory synapses, 
weights of the former are modified by the Willshaw-type learning rule. Simulations of the 
memory function of the network are performed for sparse coding with the low firing activity 
(/ <^ 1) because neurophysical evidences indicate that only a small fraction of neurons are 
active at a given time Our simulations demonstrate that our HH neuron networks 



works fairly well in memory function within the temporal coding. 
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Our paper is organized as follows: In the next Sec. II, we describe a neural system 
consisting of HH neurons which are fully connected with time-delayed couplings. The 
results of simulations are presented in Sec. III. The retrieval of the stored patterns by perfect 
and imperfect input patterns are studied in Sec. 3.1 and Sec. 3.2, respectively, and the effect 
of the distribution of the time delay in couplings is discussed in Sec. 3.3. The final Sec. 
IV is devoted to discussions concerning the relevant rate-code model and the variability of 
interspike intervals (ISI) of output spike trains. 

II. ADOPTED NETWORK MODEL 

We adopt a network of HH neurons which stores and retrieves sparsely coded P 
patterns. HH neurons described by identical parameters, are fully coupled by synapses with 
time-delayed couplings. The input information is assumed to be stored in the synaptic 
plasticity as will be discussed shortly [Eq.(ll)]. 

Dynamics of the membrane potential Vj of the HH neuron j (= 1, .., A^) is described by 
the non-linear delay-differential equations given by 

C dVj{t)/dt = -I'^^'iVj, m„ hj, rij) + If* + If\ (4) 

where C = 1 /iF/cm^ is the capacity of the membrane. The first term of Eq.(4) expresses 
the ion current given by 

/f "^(V;-, m„ h,, nj) = g^.m]h,{V, - V^^) + qku^V, - Vk) + gi.{V, - Vi^. (5) 

Here the maximum values of conductivities of Na and K channels and leakage are (/Na = 
120 mS/cm^, (^k = 36 mS/cm^ and gi, = 0.3 mS/cm^, respectively, and the respective 
reversal potentials are \4[a = 50 mV, Vk = —77 mV and Vi^ = —54.5 mV. The dynamics 
of gating variables of Na and K channels, rrij, hj and rij, are described by linear-differential 
equations, whose explicit expressions have been given (for example see ||25|| ). 
The second term in Eq.(4) denotes the external input current given by 

If = 9syn {Va - ^ a{t - Un), (6) 

n 

which is induced by the pre-synaptic spike-train input applied to the neuron i, given by 

Ui{t)=V,Y.^it-tin)- (7) 

n 

In Eqs.(6) and (7), is the n-th firing time of the spike-train inputs, Qsyn and Vc denote the 
conductance and the reversal potential, respectively, of the synapse, Tg is the time constant 
relevant to the synapse conduction, and a(t) is the alpha function given by 

a{t) = it/n) e-*/- Q{t). (8) 

where G(t) is the Heaviside function. 

When the membrane potential of the j-th neuron Vj{t) oscillates, it yields the spike-train 
output, which may be expressed by 
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Uoj{t) = V^Y. ^(t-tojm), (9) 
m 

in a similar form to Eq.(7), tojm being the m-th firing time when Vj{t) crosses = 0.0 mV 
from below. 

The third term in Eq.(4) expresses the interaction between neurons: 

If = G( E E i9]T - 9]t){Va - K) a{t - T,, - U™)). (10) 

with 

9Tk=9e.cQ{j:ifit^), (11) 

G{x)=xQ{x), (12) 

In Eqs.(10)-(12) the self-interaction (j = k) is forbidden, ^fl^"^ denotes the conductance of the 
excitatory synapse, g^Jll' that of the inhibitory synapse, and the time delay Tjk (= Tkj = Td) 
is the sum of conduction times for currents from the neurons k to j through the axon and 
dendrite. The excitatory synaptic strength g^^'^ in Eq.(ll) takes either the value of or g^xc 
after Willshaw et al. The synaptic plasticity occurs only when both the pre- and 

post-synaptic neurons fire simultaneously, in accord with biological findings. The uniform 
inhibition g'^Jl!^ = ginh is included to reduce the cross-talk noise. 

The function G{x) in Eq.(12) represents the dendrite processing of postsynaptic currents, 
and its uni-directional form is adopted such that neurons do not fire for resultant inhibitory 
inputs It is well known that an HH (and FN) neuron can fire even for an inhibitory input 
with the inhibitory rebound process [^, whereas an IF neuron cannot. Figure 1(a) and 
1(b) show responses of a single HH neuron to excitatory and inhibitory inputs, respectively, 
which are calculated by setting Jj"* = in Eq.(4). With an application of an excitatory 
input, a HH neuron is depolarized to fire at 2.8 msec after the trigger, and it recovers to the 
rest state with the refractory period of about 10 msec. When an inhibitory input is applied, 
on the other hand, a HH neuron is once hyper-polarized and then depolarized to fire after 
14.6 msec of the input injection. Because the firing of neurons for inhibitory inputs is not 
appropriate for the memory retrieval, we introduce the uni-directional function G{x) given 
by Eq.(12), which suppresses the negative (inhibitory) input, related discussions being given 
in Sec. IV. 

Patterns to be stored are expressed by a vector of = {^Y^ \ j = 1 — N} where 
H = 1—P and = 0(1) for the quiescent (firing) state of the neuron j. We randomly create 
the patterns to be stored such as to satisfy the condition for a given number of unit-codes, 
M(= fN): 

M = M^^"^ =Y,if . (13) 
i 

Without a loss of the generality, we take for the /i = 1 pattern that = 1 for j < M and 
^j^^ = for j > M. Simulations have been repeated for ten samples, each of which includes 
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P patterns. The standard deviations in the calculated results are shown by error bars in 
figures of the following sections. 

The retrieval of the first pattern of ^^^^ is made by injecting a proper input pattern of 
C to our network at t = 0. The input pattern may be C = ^^'^^ (perfect input) or ^ 7^ ^^^^ 
(imperfect input). Differential equations given by Eqs.(4)-(12) are solved by the forth-order 
Runge-Kutta method by the integration time step of 0.01 msec with double precision. We 
assume that all HH neurons are in the same initial conditions given by 

Vj{t) = -65 mV, mj{t) = 0.0526, hj{t) = 0.600, nj{t) = 0.313, for j = 1, 2 at t = 0, (14) 

which are the rest-state solution of a single HH neuron (Jj"* = 0). The initial function for 
Vj{t) of all HH neurons is given by 

Vj{t) = -65 mV for j = 1, 2 at t G [-r^, 0). (15) 

The adopted model parameters are summarized as follows: = 100, Va = 30 mV, Vc = 
—50 mV, Td = 10 msec, Ts = 2 msec ||25|, Qsyn = Qexc = 0.3 mS/cm^ and Qmh = 0.24 mS/cm^. 



The time delays are taken to be a single constant value of Tjk = Tkj = Td in Sec. 3.1 and 3.2, 
while in Sec. 3.3 they are allowed to uniformly distribute in the limited range. We choose 
Td = 10 msec such that it not only satisfies the condition of > 7 msec, which arises from 
the absolute refractory period of a HH neuron [Fig. 1], but also it is witnin the biologically 
conceivable values [1^ . A choice of gexc and gmh will be discussed shortly. Although the size 
of A^ (and P and M) is not sufficient large because of a limitation of our computer facility, 
the calculated results are expected to be useful and meaningful. 



III. RESULTS OF SIMULATIONS 
3.1 Retrieval by perfect inputs 

Firstly we consider, in this subsection, the case in which ^ = ^^^^ is injected for retrieval 
of ^^^^ (the perfect input): the retrieval by imperfect-input injection (^ 7^ ^^^^) will be 
discussed in the following subsection 3.2. 

We define the initial overlap, m^, between the input pattern ^ and the pattern to be 
retrieved ^^^^i 

m, = {l/N)Y^[2^f-l][2Q-l]. (16) 
j 

It is easy to see that rrii = 1.0 for Q = Similarly the time- dependent overlap, m{t), 
between the state of the system and the pattern is defined by 

m{t) = {l/N)Y.[2^^'^-l][2vAt)-l], (17) 
j 

where rjjit) is given by 

7]j{t) =1, if Uoj = 1 at t e [tojm - St, tojm + St] , 

= 0, otherwise, (18) 
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tojm ijn = 1,2,..) being the firing time of the neuron j and St (= 2.45 Tsyn ~ 5 msec) 
denoting the margin with a magnitude of a half-width of the alpha function given by Eq.(8). 
We hereafter regard firings with the overlap of m(t) = 1.0 at t ~ 50 as a successful 
retrieval. 

An example of an retrieval of of ^^-^^ for a given set of patterns, is shown in Fig. 2. 
We show the time course of a trigger input Uij, the output Uoj, the pre-synaptic current 
Ij (= J|^'* + Jj"*) and the membrane potential Vj of the neuron of j = 1, for an injection 
of the input pattern <^ at t = with a = 0.30, / = 0.1, gsys = Qexc = 0.30 and ginh = 
0.24 mS/cm^. The pre-synaptic current Jf^'* starting from t = is yielded by the trigger 
input Uij, while other pre-synaptic currents, for example, starting from t = 12.6 msec, arise 
from the interaction term /j"* through the couplings. The firing of the neuron j = 1 is given 
by Uoi(t) = 1, and it is compactly depicted in Fig. 3(a), where dots express firings of neurons 
numbered by the index j = 1 — 100 in the vertical scale as a function of time [Uoj{t) = 1 in 
Eq.(9)]. Figure 3(a) shows a successful retrieval for a = P/N = 0.30 with / = M/N = 0.10, 
gexc = 0.30 mS/cm^ and A = 0.8. The time-dependent overlap m{t) given by Eq.(17) is 
always unity: = m(t) = 1.0 at t > 0, which is calculated at t = toim ("^ = 1, 2, ..), firing 
times of the neuron 1, and which is plotted at the upper part of Fig. 3(a). The period 
of firings To is determined by To = Td + Tr ~ 12.5 msec where denotes the delay time 
of couplings and ~ 2.5 msec is the response time of a HH neuron. Figure 3(b), on the 
contrary, shows a failed case for a larger a value of a = 0.50, yielding extra firings of neurons 
of j > 10 and non-firings of j < 10. We note that rrii = m{t) = 1.0 at t = but m{t) quickly 
deviates from its initial value at t > 0. Figures 3(a) and 3(b) show that a single trigger 
impulse is sufficient to recall the stored pattern. 

In order to show the importance of the uniform, inhibitory synapse introduced by ginh 
in Eq.(lO), we plot, in Fig. 4, the A (= ginh/ gexc) dependence of the critical storage capacity 
«c = -Pmax/A^, above which the retrieval of the input pattern cannot made. We note that 
ttc decreases as the value of A decreases. Although the A dependence of etc with gexc = 
0.3 mS/cm^ is slightly different from that with g^xc = 0.5 mS/cm^ for / = 0.06, the result 
with gexc = 0.3 mS/cm^ is identical with that of gexc = 0.5 mS/cm^ for / = 0.10. We have 
decided to adopt gexc = 0.3 mS/cm^ and A = 0.8 {ginh = 0.24 mS/cm^) for our simulations. 

The calculated storage capacity (y.Q cis db function of the mean activity / is plotted in 
Fig. 5, where bars express the standard deviations due to the sample dependence. It clearly 
shows that ttc increases as / decreases. Theoretical analysis based on the Hopfield model 
has shown that the storage capacity diverges as etc oc (—1// log/) in the limit of vanishing / 



|T| . Recent calculations of oscillating associative memory made by Aoyagi and Nomura 
||T9| also support the (—1// log/) dependence. Although the / values of our simulations for 
spiking HH neuron networks are not sufficiently small, our result seems to support it: the 
dotted curve expresses 10/(MlogM), which is proportional to (—1// log/) in the limit of 
/-O. 

3.2 Retrieval by imperfect inputs 

Next we discuss the retrieval of ^^^"^ when an imperfect input, ^ ^ is injected. 

We assume that an input pattern ^ is similar to ^^^^ but some of its codes are modified, 
keeping unit-code number unchanged: M = J2jCj- We adopt C whose codes are same as 
those of ^^-"-^ except 1-codes in j E [M — AM/2, M)] are changed to 0-codes, and 0-codes in 



7 



j G [M + 1, M + AM/2] to 1-codes , where even AM (> 2) is the number of changed codes. 
The initial overlap of <^ thus created is mj = 1.0 — 2 AM/N . 

Figure 6(a) shows an example of the successful retrieval when an input with the initial 
overlap of = 0.84 is injected for a = 0.10 and / = 0.20. The time dependence of m{t) 
plotted in Fig. 7, shows that m{t) = rrii = 0.84 just after an input injection and m{t) 
gradually approaches unity. The time dependence of m{t) for the case of rrii = 0.92 also 
successfully approaches unity. On the contrary, when an input pattern has a smaller initial 
overlap of mi = 0.68, the retrieval is failed; firings of neurons spread all over the system as 
shown in Fig. 6(b) and the overlap m(t) decreases from the initial value of rrii = 0.68 as 
shown in Fig. 7. The failed case is plotted also for rrii = 0.52, whose transient value of m{t) 
is not necessary smaller than that for rrii = 0.68. 

The dashed curves in Fig. 8 show the storage capacity etc against the initial overlap 
rrii for / = 0.20. It shows the critical, initial overlap is about rrii = 0.73 for a = 0.1 and 
/ = 0.20 (Fig. 7). We note in Fig. 8 that for / = 0.20, the retrieval is effectively possible 
for rrii ~ 0.6. On the contrary, for / = 0.10, the retrieval is possible only in the restricted 
region of 0.92 < < 1.0. Figures 7 and 8 reminds us the similar calculation made by 
Amari and Maginu for the Hopfield model [^]. 

3.3 Effect of distributed time delays 

So far we have assumed that the time delays are the same for all couplings connecting 
neurons: Tjk = Tkj = Td- It is expected to be not the case in real biological systems. In order 
to study the effect of their distributions, we assume that time delays uniformly (randomly) 
distribute as 

Tjk = Tkj G [Td - Ar/2, Td + Ar/2], (19) 

where At is a width of the time-delay distribution. 

When the width of the time-delays distribution becomes large, it is difficult for post- 
synaptic currents to sustain firings of neurons. The solid curve in Fig. 9, expressing the 
critical storage ac for / = 0.06 as a function of At for the perfect-input retrieval, shows that 
the critical storage decreases when the width of the distribution becomes wide, and that ac 
vanishes for At > 6 msec. In the case of / = 0.10, whose result is plotted by dashed curve, 
ac vanishes for At > 9 msec. Although the time- dependent overlap of / = 0.10 deviates 
from 1.0 for At > 9 msec, the oscillating firings may continue with spread firing times. Fig. 
10 shows such a case of At = 10 msec, in which neurons from j = 1 to 10 fire as the pattern 
^^■"■^ but not in a coherent way. The time dependent overlap defined by Eq.(17) oscillates 
with 0.88 < m{t) < 1.0, as is shown in the upper part of Fig. 10. 

The critical width of the time-delay distribution, Atc, above which the retrieval of the 
stored patterns is failed, depends on the average time delay of r^. Figure 11 shows Ar^ for 
three cases of (/, a) = (0.06, 0.30), (0.06, 0.10) and (0.10, 0.10). We should note that the 
time delay has the lower bound of 7 msec below which the retrieval is not possible because 
of the absolute refractory period of a HH neuron [Fig.l], while it has no upper bounds. For 
both q;=0.30 and 0.10, the critical width for / = 0.06 has a maximum value of Atc = 13 
msec at Td = 15 msec, and Ar^ ~ 8 msec at > 25 msec. The critical width for / = 0.10 
is Atc = 13 msec at Td ~ 15 msec, which is larger than that for / = 0.06. Our HH neuron 
network is fairly robust against the distribution of the time delays in couplings. 
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IV. DISCUSSION AND CONCLUSION 



Our simulations have demonstrated that neural networks consisting of spiking HH neu- 
rons may show a fairly good performance as an associative memory with the memory capacity 
of ~ 0.4 — 2.4 for / ~ 0.04 — 0.10 [Fig. 5]. It is worth to compare the storage capacity 
of our HH neuron networks with that of the rate-code model in which the dynamics of the 
firing rate Xj{t) is given by 

x,{t +l) = Q{Y. i^jk - v) Xk{t) - 9). (20) 

In Eq.(20) Wjk is determined by the Willshaw-type rule given by Eq.(2), and z/ (0 < < 1) 
and 9 are the inhibitory coupling and the threshold, respectively, which are introduced to 
suppress cross-talk noises ||2^- The storage capacity depends on /, z/ and 9. The 
calculated Oc of this rate-code model is shown as a function of / by the solid curve in Fig. 
12, where parameters ofA^ = 100,z/ = 0.8 and ^ = 0.5 are adopted [^]. We get the memory 
capacity of etc ~ 0.4 — 3.7 for / ~ 0.04 — 0.10, which is comparable to that of our HH neuron 
network. 

For a comparison, we have repeated calculations of the storage capacity of the rate- 
code model mentioned above, employing the conventional Hebb-type rule given by Eq.(2) 
instead of the Willshaw-type one given by Eq.(3). The calculated / dependence of the 
memory capacity is plotted by the dashed curve in Fig. 12. The storage capacity becomes 
Uc ~ 0.2 — 1.5 for / ~ 0.04 — 0.10, which is about a half of the values in the rate-code model 
with the Willshaw-type synapses. The critical storage capacities in the Hopfield model with 
Willshaw-type [Eq.(2)] and Hebb-type synapses [Eq.(3)] increase as the activity / decreases, 
which is in agreement with the theoretical analysis |^8||- |^ . 



We now examine the following two assumptions adopted in Eq.(lO) for the interaction 
term in our HH neuron network: 

(i) the Willshaw-type rule for weights of excitatory synapses and the uniform inhibitory 
synapses [Eq.(ll)], and 

(ii) the uni-directional function, G{x) [Eq.(12)] for dendrite processing. 

We have employed the assumption (i) to enhance the storage capacity. In order to show this 
explicitly, we calculate the / dependence of the storage capacity when the Willshaw-type 
learning rule given by Eq.(ll) in our HH neuron network is replaced by the conventional 
Hebb-type one: 

g,k = ge.c E d"^. (21) 

The calculated storage capacity is plotted by the solid curve in Fig. 13 as a function of 
the average activity /. The memory capacity is Uc ~ 0.03 — 0.14 for / ~ 0.04 — 0.10, which 
is much smaller than the corresponding value of Uc ~ 0.4 — 2.4 obtained with Willshaw-type 
rule [Fig. 5]. This clearly shows that the storage capacity with the Willshaw-type rule 
[Eq.(ll)] is enhanced by a factor of ten. The reason why the solid curve in Fig. 13 has a 
discontinuous change at / ~ 10, is not clear at the moment. The solid curve loosely follows 
the dotted curve expressing 1/(M logM), which is proportional to (—1// log/) at / — *• 0. 
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The assumption (ii) on G{x) has been introduced to prevent firings for inhibitory (neg- 
ative) inputs. This function G{x) may be regarded to mimic the inward rectification of 



dendrites which make the response of a neuron to hyperpolarizing outward currents 
much worse than that to depolarizing inward currents. If the uni-directional function: 
G{x) = X 0(x) given by Eq.(12) is replaced by the linear one: G{x) = x, HH neurons 
fire for both excitatory and inhibitory inputs, and then the neural networks cannot retrieve 
the stored patterns. In this case, the weight of inhibitory synapses, gjjf^, in Eq.(lO) is 
properly determined by a learning rule depending on the stored patterns, as Lytton 
has proposed for the feedforward HH network. Such a procedure, however, yields a poor 
performance as an associative memory working only in a narrow parameter range |12|. It 



would be necessary that the assumptions (i) and (ii) adopted in our HH neuron network are 
examined and sought physiologically. 

It has been reported that a large variability (c^, = 0.5 ~ 1.0) is observed in spike trains 
of non-bursting cortical neurons in visual VI and MT of monkey HH], which is in strong 



contrast with a small Cy (= 0.05 ~ 0.1) in motor neurons [0. There have been much 
discussions how to understand the observed large variability |^- |Q: a balance between 



excitatory and inhibitory inputs ||4^, the high physiological gain in the plot of input current 



vs. output frequency correlation fluctuations in recurrent networks |^, the active 



dendrite conductance |50|, input ISIs with the distribution of a slow- decreasing tail [51|, and 



input ISIs with large [|25| ||52|. Based on our simulations, we discuss the behavior of the 
interspike interval (ISI) of output spike trains which is defined by 

where tojm {fn = 1,2,..) is the firing time of the neuron j [Eq.(9)]. Histograms shown in 
Figs. 14 (a)- 14(c) express the distribution of ISIs of all HH neurons for < t < 500 msec. 
Figure 14(a) denotes the distribution of output ISIs of a successful retrieval with the initial 
overlap of rrii = 0.84, which have been discussed in Sec. 3.2: the time course of its firings of 
neurons is plotted in Fig. 6(a) with parameters of / = 0.20, a = 0.10 and = 10 msec. The 
mean and root-mean-square (RMS) values of the output ISI are fio = 11.98 and cto = 1.42 
msec, which yield the dimensionless variability of c^, = (Xo/fio = 0.12. The histogram of 
Fig. 14(b), on the other hand, shows the distribution of ISI for a failed retrieval case with 
a smaller initial overlap of rrii = 0.68: the time course of neuron firings are plotted in Fig. 
6(b). We note that ISIs distribute up to 30 msec, obtaining /Xo = 13.98, Uo = 5.69 msec, 
and Cy = 0.41. The histogram of Fig. 14(c) expresses also a failed case with more smaller 
rrii = 0.56, which leads to /io = 16.17, Uo = 17.76 msec and c^ = 1.10. Note that there is 
a small distribution of ISI even at 30 < To < 60 msec, where plotted result is magnified by 
a factor of ten. Repeating these calculations, we obtain, in Fig. 15, the rrii dependence of 
fio, o'o and Cy (for / = 0.20, a = 0.10 and = 10 msec). For nii > 0.9, we get /Iq = 11.76 
and o"o ~ msec, then c^ ~ 0. When rrii is more reduced, /Iq is increased because ISIs with 
To > 20 msec appear, which make Uo and c^ increase. Figure 15 shows that Cy is larger 
for smaller rrii, and that the variability becomes considerable (c„ ~ 0.5) even for successful 
retrievals with rrii > 0.7 [Fig. 6]. The situation is similar when a retrieval is worsen because 
of spread distributions of time delays in couplings. For example, in the case having been 
shown in Fig. 10, where we plot the time course of firings for Ar = 10 msec with the 
parameters of / = 0.10, a = 0.10, = 10 msec and rrii = 1-0, a considerable value of 



10 



Cv = 0.45 is obtained which arises from /Iq = 18.61 and Uo = 8.31 msec. Our simulations 
show that the variabihty of ISIs may be considerable while neurons are trying to retrieve 
the stored memory. We hope that results of our simulations might have some relevance to 
the observed large variability. 

To summarized, taking account of the two issues on the Hopfield-type neural networks 
raised in the Introduction, we have proposed the biologically plausible network consisting 
of spiking HH neurons with the Willshaw-type synapses, whose simulations demonstrate its 
storage capacity to be comparable to that of the rate-code network. We may extend and 
modify our model into various directions. For example, it is straightforward to apply our 
method to the feedforward hetero-associative memory of temporal-code Anderson-Kohonen- 
type networks. When the excitatory synapse given by Eq.(ll) is changed as 

^ir = ^?e.c e(E er'^ d"^ (23) 

the patterns of yU = 1 — P can be retrieved in a temporal sequence. This shows a pos- 
sibility of the storing and retrieval of temporal patterns. The synapse given by Eq.(23) 
corresponds to the asymmetric coupling in the Hopfieled model [^]. It has been shown 



that the Hopfieled model with asymmetric couplings shows intrigue dynamics like chaos. 
In the present paper, we have taken no account of noises. Quite recently, Kanamaru and 
Okabe |]TB|] have investigated the effect of noises on an associative memory of the FN neuron 



network, showing that a retrieval of the stored patterns may be improved by noises, just like 
the stochastic resonance. It would be the case also in our HH neuron network. We cannot, 
however, draw any definite conclusion until performing simulations because their leaning 
rule given by Eq.(l) with a = and 6 = / is different from ours given by Eq.(ll). It is well 
known that temporal-code neurons are more efficient in data processing than the rate-code 



neurons because the former can carry much information than the latter |54]. It is expected 
to be true also in the memory function. We suppose that there could be alternative, more 
efficient mechanisms, by which real temporal-code networks might be operating for learning 
and memory. 
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FIGURES 

FIG. 1. Responses of the membrane potentials, V (solid curve), of a HH neuron to (a) 
excitatory and (b) inhibitory external currents, /^^* (dashed curve): a HH neuron fires not only 
by excitatory input but also by inhibitory inputs. 

FIG. 2. The time course of (a) input Un, (b) output Uoi, (c) the post-synaptic current 
/i (= /f^* + 11""^) and (d) the membrane potential Vi of the neuron j = 1 when an input pattern 
is injected at t = with a = 0.3, / = 0.10, gexc = 0.30 mS/cm^ and A = 0.8; the vertical scale is 
for Vi only, and /I"*, Uoi and Uu are shown in arbitrary units. 

FIG. 3. The time course of firings of neurons for (a) a = 0.30 and (b) a = 0.50 with / = 0.10, 
dexc = 0.30 mS/cm^ and A = 0.8; dots express firings of neurons numbered by index shown in the 
vertical scale: the upper part shows the relevant overlap of m{t). 

FIG. 4. The storage capacity function of A (= gmh/gexc) for / = 0.06 and 0.10 with 

dexc = 0.3 mS/cm^ (solid curve) and gexc = 0.5 mS/cm^ (dashed curve). 

FIG. 5. The storage capacity Uc as a function of the average activity /. The solid curve 
denotes the calculated results and the dotted curve expresses 10/ (M logM), error bars denoting 
the standard deviations for ten samples. 

FIG. 6. The time course of neuron firings for an injection of imperfect input patterns with 
initial overlaps of (a) = 0.84 and (b) rrii = 0.68 for / = 0.20 and a = 0.10. 

FIG. 7. The time dependence of the overlap m{t) for various rrii with / = 0.20 and a = 0.1. 

FIG. 8. The storage capacity ac as a function of the initial overlap for / = 0.10 (solid 
curve) and / = 0.20 (dashed curve). 

FIG. 9. The storage capacity function of a width of the time-delay distribution, At, 

for Td = 10 msec. The solid and dashed denotes the calculated results for / = 0.06 (solid curve) 
and / = 0.10 (dashed curve), respectively. 

FIG. 10. The time course of firings of neurons for / = 0.10, a = 0.10, At = 10 and = 10 
msec: the upper part shows the relevant overlap of m{t) (see text). 

FIG. 11. The critical width, Atc, of the time-delay distribution as a function of for 
(/, a) = (0.06,0.30) (solid curve), (0.060, 0.10) (dotted curve) and (0.10, 0.10) (dashed curve). 
Calculations are made with a step of At = 1.0 msec for a given t^. 
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FIG. 12. The storage capacity etc as a function of the average activity / of the rate-code neuron 
network with the Willshaw-typc synapses (sohd curve) and the Hebb-type synapses (dashed curve) , 
the dotted curve denoting 10/(M logAf) (see text). 

FIG. 13. The storage capacity ac as a function of the average activity / of the HH neuron 
network with the Hebb-type synapses [Eq.(21)]. The sohd curve denotes the calculated result and 
the dotted curve expresses 1 / (M logM) . 

FIG. 14. Histograms of ISIs of spike-train outputs in a retrieval process with the initial overlap 
of (a) rrii =0.84, (b) 0.68 and (c) 0.56 for / = 0.20, a = 0.10 and r^i = 10 msec, the histogram at 
To > 30 msec of (c) being enlarged by a factor of ten. The time course of neurons firings for (a) 
and (b) are plotted in Fig. 6(a) and 6(b), respectively. 

FIG. 15. The average (/Xq, thin solid curve) and RMS values of ISIs (cTo, dashed curve), and 
the variability (c„, bold solid curve) as a function of the initial overlap (mj) for / = 0.20, a = 0.10 
and Td = 10 msec, the left (right) vertical scale being for fj,o and o"o (c^). Error bars denote the 
standard deviation of c^, and those for fig and cJo are not shown for illegibility of the figure. 
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